Freeway is excluded from this table as Junyent et al. [ Epochs 8 Loss Function for Policy Categorical crossentropy Loss Function for Value Function Huber Discount factor used in TD Learning 0.99 Time steps between target network updates (for value network) 10,000 Interval size of learning schedule Due to computational restraints we could not tune the hyperparameters of N-CPL.

We propose new width-based planning and learning algorithms inspired from a careful analysis of the design decisions made by previous width-based planners. The algorithms are applied over the Atari-2600 games and our best performing algorithm, Novelty guided Critical Path Learning (N-CPL), outperforms the previously introduced width-based planning and learning algorithms $\pi$-IW(1), $\pi$-IW(1)+ and $\pi$-HIW(n, 1). Furthermore, we present a taxonomy of the Atari-2600 games according to some of their defining characteristics. This analysis of the games provides further insight into the behaviour and performance of the algorithms introduced. Namely, for games with large branching factors, and games with sparse meaningful rewards, N-CPL outperforms $\pi$-IW, $\pi$-IW(1)+ and $\pi$-HIW(n, 1).